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(54) Automatic image layout method and system 



(57) A method for automatically generating tem- 
plates to define how different images should be ar- 
ranged on a page. Through the application of standard 
optimization techniques with cost functions tuned to the 
needs ol SmartPlaten technology, a collection of digital 
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images are arranged on contact sheet-like templates 
whereon each image is scaled and positioned to mini- 
mize the amount of white space on the page. Resulting 
images are as large as possible, yet fit within a pre-des- 
ignated page area. 
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Description 

Field of the Invention 

[0001] This invention is generally related to layout 
techniques for digital images and, more particularly, to 
an automatic image layout technique which allows a col- 
lection of digital images to be formatted in a single con- 
tact sheet-like layout for rapid viewing and cataloging. 

Background of the Invention 

[0002] Computer based document handling systems 
are generally divided into four broad categories: text ed- 
itors and word processing systems; formatters; syntax 
directed editors; and specialized tools. Most systems 
have features from more than one of these four broad 
categories in addition to any image processing neces- 
sary for editing images. An image processor reads an 
image of a document using an image input unit, stores 
it in a memory in the form of image data, and subjects 
the stored image data to editing operations such as ad- 
dition and deletion. As one of the editing operations, part 
of the read document is electronically cut and pasted to 
a predetermined place in another document. In such an 
electronic cutting and pasting operation, the document 
image to be cut and the destination document image are 
displayed either on a single display unit or on separate 
display units, and a cutting image portion in the docu- 
ment to be cut and a pasting place in the pasting (des- 
tination) document are similarly specified by a mouse, 
for example. In the related art image processor, the cut- 
ting image portion and the pasting place are specified 
by the operator by selecting appropriate positions while 
looking at the screen. For this reason, it is not possible 
to cut or paste figures precisely. Particularly, shifts in po- 
sition are noticeable when the same patterns must be 
juxtaposed. US Patent No. 5,224,181 toTsutsumi enti- 
tled "image processor" and US Patent No. 5,202,670 to 
Oha entitled "image Processing Apparatus n both dis- 
cuss image processing generally. For documents in 
which presentational considerations are important, the 
documents must be submitted to a formatter for prepa- 
ration prior to presentation. Formatters are non-interac- 
tive tools that process a document to produce either a 
display independent or a device dependent layout spec- 
ification. Documents are submitted to formatters in the 
form of descriptions on file and they carry out the 
processing and return the overall results after a certain 
period of time. High-level formatters work on the basis 
of a logical description of the document. The user is not 
required to specify the presentation details desired. The 
user deals with the logical organization of the document, 
i.e., the different types of elements that appear in the 
document, such as, for example, section, paragraph, 
heading, summary, etc. The formatter handles the lay- 
out presentation of these elements. Low-level format- 
ters make it possible to include commands within the 



document description to enable changes in other char- 
acteristics of the document, such as, for example, font, 
spacing, margins and justification. US Patent No. 
5,438,512 to Mantha et al. entitled "Method and appa- 

s ratus for specifying layout processing of structured doc- 
uments" is directed to high-level formatters. Mantha dis- 
closes a method and apparatus for specifying layout 
processing of logically structured documents in compu- 
ter document handling systems. The Mantha method 

to and apparatus allow the specification of the generic log- 
ical structure of the structured document in terms of re- 
lational attribute grammars. 

[0003] Most interactive systems allow the user to see 
the layout of a document as it is being prepared. These 

is interactive systems also separate the logical structure 
of the documents from the specification of the presen- 
tation details. Typically, interactive systems as well as 
high-level formatters, use a grammatical notation to de- 
scribe the logical structure of documents. These logical 

20 structures are mostly hierarchical in nature and tree 
structures are used to represent them. 
[0004] The need for document processing devices 
which can generate a layout structure of a document by 
applying a template (i.e., rules for layout) to a logical 

25 structure of the document are apparent with the wide 
use of microprocessors to process and properly render 
electronically created or digitally scanned documents, 
or images as more specifically referred to at times. In 
general, both of the logical development and the layout 

so of an actual output document are important factors of 
the document. However, the layout is not necessarily im- 
portant in the process of producing a document. For ex- 
ample, although the layout is necessary once the con- 
tents of a document are determined, in some cases the 

35 layout is not considered in the initial stage of document 
production where the logical development is not clear 
yet. Furthermore, different layouts may be required for 
one logical development in some cases; for instance, 
when the same document needs to be distributed to a 

40 plurality of persons or sections. Recognizing the above 
distinction, there have been proposed techniques of 
generating a layout representing such structures called 
"layout structure" from a structure called "logical struc- 
ture" which represents the chapter construction of a 

45 document. The process of generating a layout structure 
from a logical structure is called a layout process, and 
is performed by a program (layout processing program) 
that is incorporated in document processing systems. 
[0005] In order to generate various forms of layout 

so structures from the same logical structure, the operation 
of a layout processing program needs to be modified. 
However, in general, a program itself cannot be modified 
properly by ordinary users because the modification 
needs expert knowledge. For this reason, there has 

55 been employed a technique of altering a program by us- 
ing parameters such as "page size is A4 B and "double 
column setting" for textual documents. However, ac- 
cording to this technique, the kinds of specifiable param- 
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eters are limited to ones that are preset in the program. 
Furthermore, to control the operation of a layout 
processing program having many functions it is neces- 
sary to specify a large number of parameters which pre- 
cludes ordinary users from using such a program. To 
solve the above problems, there have been proposed 
techniques of controlling a layout processing program 
by using, rather than simple parameters, a data struc- 
ture representing a layout template. One of those tech- 
niques is a "generic layout structure" prescribed in the 
international standards "ODA" (IS08613, Information 
Processing-Text and Office Systems-Office Document 
Architecture (ODA) and Interchange Format (1989)). 
The ODA only sets forth data structures for representing 
document structures and guidelines for their use, and 
does not describe actual layout processes. However, it 
is apparent that the following functions are needed to 
perform layout operations according to the ODA. The 
layout process having the following functions is herein- 
after referred to as "ODA layout process. ■ 



The functions (mechanisms) (1)-(4) are not necessarily 
effected individually, but could be combined when de- 
sired. Combining the functions can improve the efficien- 
cy of the layout process. 

[0006] US Patent No. 5,381,523 issued to Hayashi 
entitled "Document processing device using partial lay- 
out templates" discloses partial layout templates that 
are prepared for respective partial logical structures of 
a hierarchical logical structure of a document. Each par- 
tial layout template expresses rules for producing a lay- 
out of the corresponding partial logical structure. A par- 
tial layout generator produces a partial layout structure 
by recursively calling itself or by calling a content layout 
system while referring to the partial layout template, and 
pours the generated partial layout structure into a low- 
est-rank frame. A layout of the entire logical structure is 
produced by sequentially performing the partial layout 
operations. Hayashi also only relates to a document 
processing device which can generate a layout structure 
of a single document by applying a template (i.e., rules 
for layout) to a logical structure of the single document. 
[0007] With the growing interest in digital photography 
and the necessity to lay out and view several digital im- 
ages as a document, a need still exists for a document/ 
image structuring process which would allow a micro- 
processor to organize the layout of several images on 
a sheet-like medium. Needed is a technology that ena- 
bles a user to gang scan (several images on the scanner 
platen at once) many images and automatically locate 
each image, crop them, and correct any rotation errors 



associated with each image. Resulting images can then 
be stored individually or as a structured image with a 
user-defined or simple row-column layout which can be 
created. A detailed description of Structured Images is 
s provided in US-A 5,485,568 issued to Venable et al. on 
January 16, 1996. entitled "Structured Image (SI) format 
for describing complex color raster images." 
[0008] There is a need for technology that will auto- 
matically generate the digital equivalent of a photogra- 
phers contact print sheet in which the segmented imag- 
es will be scaled and distributed on a resulting page 
such that each image is scaled as large as possible 
wherein there is a minimum of white space remaining 
on the rendered page. 

[0009] As a first constraint to adequately addressing 
such a need, all images must be scaled by the same 
amount, i.e., the relative size of each segmented image 
stays the same. A second constraint should be in form- 
ing the equivalent of a bounding box with minimum white 
space (space uncovered by an rectangle). Once a so- 
lution to the above constraints are found, many images 
may be scaled by the appropriate amount to make a 
bounding box the same size as the printable area of pa- 
per. It is a feature of the present invention to effectively 
address the constraints in order to accomplish the de- 
sired results described above. 
[0010] Other advantages and salient features of the 
invention will become apparent from the detailed de- 
scription which, taken in conjunction with the drawings, 
disclose the preferred embodiments of the invention. 

Summary of the Invention 

[001 1 ] Disclosed is the technology that enables a user 
to "gang scan" (to scan several images on the scanner 
platen at once) many images and automatically locate 
each image, crop them, and correct any rotation errors 
associated with each image. Resulting images can be 
stored individually or as a structured image with a user- 
defined or simple row-column layout which can be au- 
tomatically created with a microprocessor based sys- 
tem. The invention enables the system to automatically 
generate the digital equivalent of a photographers con- 
tact print sheet onto which the segmented images will 
be scaled and distributed such that each image on the 
resulting page is scaled as large as possible and where- 
in there is a minimum of white space remaining on the 
rendered page. Furthermore, the images are scaled by 
the same amount, i.e., the relative size of each segment- 
ed image stays the same. The optimum spatial distribu- 
tion of the images, represented by rectangles, are de- 
termined under the following guidelines/constraints: 

1) rectangles (representing each image) do not 
overlap, 

2) the bounding box of all rectangles has a minimum 
area, 

3) the aspect ratio of the bounding box is a fixed 
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(1) Layout processing function based on a layout 
template (generic layout structure) 

(2) Layout process selecting function (top-down or 
bottom-up) 25 

(3) Reuse of a layout result 

(4) Category-based layout processing function 



30 



35 



40 



45 



50 



55 



5 



EP 0 929 184 A2 



6 



value R. 

[0012] The equivalent of a bounding box with mini- 
mum white space is then formed for placement of the 
images. In terms of the automatic generation of "contact 
sheets," the segmented images are the rectangles and 
the aspect ratio of the print area of the paper is R. Once 
an automatic solution to the above constraints is deter- 
mined by the microprocessor executed algorithm, the 
plurality of images (sometimes referred to as "rectan- 
gles images - throughout this disclosure) can be scaled 
by the appropriate amount to make the bounding box 
the same size as the printable area of the paper 

Description of the Drawings 

[0013] The salient features and other aspects of the 
invention will become apparent from the following de- 
tailed description of the invention when read in conjunc- 
tion with the accompanying drawings which are provid- 
ed for the purpose of describing embodiments of the in- 
vention and not for limiting same, in which: 
[0014] Figure 1 is an example of the resulting rectan- 
gle layout based on an aspect ratio of 1 .29. There are 
20 original rectangles representing images with dimen- 
sions randomly distributed between 20 and 50. The per- 
centage of white space on this "page" is 18%. 
[0015] Figure 2 illustrated the same data of Figure 1 
processed without the constraint of maintaining the as- 
pect ratio. The percentage white space is 13.2%. 
[0016] Figure 3 illustrates an additional constraint 
specified having the effect of opening the middle of the 
page and driving rectangles toward the outside edge. 
The aspect ratio constraint is included here. Percentage 
white space is 23%. 

[0017] Figure 4 illustrates the rectangle layout where 
the opposite penalty from Figure 3 is set, i.e., rectangles 
were penalized for being toward the edge. The percent- 
age white space is 28%. 

[0018] Figure 5 illustrates a block diagram of the sys- 
tem which is used to implement the invention. 

Detailed Description of the Invention 

[0019] The solution to this problem is based on com- 
binatorial minimization algorithm Simulated Annealing 
(see Numerical Recipes in C, pp. 343-352). Without go- 
ing into detail, simulated annealing is an energy minimi- 
zation algorithm useful in situations where the space of 
all possible solutions is discreet but very large such that 
it is impractical to exhaustively search the entire space 
for the absolute minimum. Simulated annealing is less 
likely to be trapped in local minima than other minimiza- 
tion algorithms. To use simulated annealing, an energy 
function that is based on the system constraints must 
be defined. At random, new configurations of the system 
are generated and the energy function for each config- 
uration is evaluated. The probability of accepting the 



new state is calculated by evaluating the function: 
Pr = exp(-(E new -E old )/kT), 

5 

where E okJ is the energy of the previous accepted state, 
E new is the energy of the new state, T is a control pa- 
rameter analogous to temperature, and k is a constant. 
A random number between (0.1 ) is then obtained using 

10 a uniform random number generator. If the random 
number is less than Pr, the new state is accepted. If E new 
is less than E 0)d , then Pr is greater than unity and the 
new state will be accepted. Therefore, new states with 
lower energies than previous states are always accept- 

15 ed. If E new is greater than E old , the new state will be ac- 
cepted with a probability given by Pr. This scheme of 
always taking downhill steps in the energy function and 
sometimes taking uphill steps is known as a Metropolis 
algorithm. 

20 [0020] In simulated annealing, the temperature T is 
periodically lowered according to an annealing sched- 
ule. The period between temperature drops is an epoch. 
The amount by which the temperature is lowered is usu- 
ally a fraction of the current value, e.g., T N+1 = T N * 0.95. 

25 The process usually begins at an initial temperature T 0 
and ends when the temperature reaches a temperature 
T end or if some other measure of convergence is satis- 
fied. This has the affect of making uphill changes less 
likely as the algorithm progresses. 

30 [0021] In our application, the positions of all rectan- 
gles define the configuration space. To generate new 
configurations from previous configurations, we define 
5 allowed operations. We do not claim these as the only 
operation possible; these are the ones we have worked 

35 with: 

1) swap the positions of two rectangles 

2) move a rectangle 

3) move a rectangle immediately to the right of an- 
40 other rectangle 

4) move a rectangle immediately to the left of an- 
other rectangle 

5) rotate a rectangle by 90 degrees. 

45 [0022] The algorithm will work using only the Move 
operation, but the use of additional operations can in- 
crease the likelihood of stepping out of local minima. 
Use the rotate operation only if the intended application 
allows the rectangles (images) to be rotated. The choice 

so of which operation is applied to generate a new config- 
uration, and which rectangles are to be adjusted, is de- 
termined by a uniform random number generator. 
[0023] The energy function must be defined such that 
function is minimized when all of the constraints are met. 

55 The energy function we define is: 

E = whiteSpace + overiapPenalty* overlap, 
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where overlapPenalty is a parameter, typically -100.0 
in our test; overlap is the area of the intersection of all 
the rectangles. The actual total overlap area is difficult 
to calculate (involving calculating the overlapping re- 
gions of a set of polygons); we use an approximation s 
that achieves the desired effect: 
overlap = 0; 

for (i=0; i<nrects-1 ; ++i) 

for (j=i+1 ; j<nrects; ++j) 

overlap += Calclntersection 
(&rects[i], &rects[j]); This code segment evaluates over- 
lap as the total sum of the intersection area between all 
rects. The "Calclntersection" procedure calculates the 
intersection area between two rectangles and the code 
segmetn loops over this procedure for all rectangles. In 
the limit where no rectangles overlap, the value of over- 
lap is zero which matches the true total overlap. In the 
limit where all rectangles overlap, the value of overlap 
is large. However, since the constraint is to minimize 
overlap, having a large overlap contribution when all 
rectangles coincide is OK. In other words, this approxi- 
mation to the total overlap does the right thing. 
[0024] WhiteSpace is a measure of the amount of 
white space left on the page in the given state. 
WhiteSpace is defined by: 

WhiteSpace = bboxArea - blackArea, 

where bboxArea is the total area of the bounding box 
(the bounding box of all rectangles adjusted to the spec- 
ified aspect ratio); blackArea is the area of the union of 
all Tectangles. blackArea is approximated by: 

blackArea = totalRectArea - overlap, 

where totalRectArea is the sum of the areas of all rec- 
tangles. 

[0025] In our work an epoch is typically 500 cycles 
with an initial temperature T Q = 100.0, and ending tem- 
perature T end = 0.01 , and fractional temperature change 
of 0.975. The simulated annealing process requires an 
initial configuration of the rectangles. In all of our exam- 
ples, the initial configuration is such that the upper left 
hand corner of all rectangles is at coordinate (0,0). 
[0026] Figure 1 is an example of the resulting rectan- 
gle layout based on an aspect ratio of 1 .29. There are 
20 original rectangles with dimensions randomly distrib- 
uted between 20 and 50. The percentage of white space 
on this ■page" is 18%. 

[0027] In Figure 2, the same data was processed but 
without the constraint of maintaining the aspect ratio. 
The percentage white space is 13.2%. 
[0028] In Figure 3 an additional constraint was spec- 
ified such that a penalty was added to the energy func- 
tion for rectangles positioned near the center of the 
page. This additional constraint has the effect of open- 



ing the middle of the page and driving rectangles toward 
the outside edge. The aspect ratio constraint is included 
here. Percentage white space is 23%. 
[0029] In Figure 4, the opposite penalty is set, i.e., rec- 
tangles were penalized for being toward the edge. The 
effect is to position rectangles closer to the center. The 
aspect ratio is not constrained. The percentage white 
space is 28%. 

[0030] It is possible to add additional constraints to 
the energy function. Some constraints may be applied 
for artistic purposes; for example, the results shown in 
Fig. 4. It may be desirable to have the rectangles aligned 
in rows or columns. This is a topic for continued re- 
search. Also, the proper normalization of the energy 
function to simplify the determination of constants such 
as overlapPenalty will be studied. 
[0031] Figure 5 illustrates a typical system 1 that 
would implement the above-described process. Within 
the computer system 2 a microprocessor 3 executes the 
formatting algorithm which is stored in memory 4. A 
scanner 5 is used to capture several images, either all 
at once from a single platen or individually. The digital 
images are then automatically organized by the micro- 
processor 2 with the inventive process recalled from 
memory 4 by the microprocessor, and the result is a con- 
tact sheet-like representation of the images which may 
also be stored in memory 4, or rendered by a printer 6. 
[0032] While the invention is described with reference 
to a particular embodiment, this particular embodiment 
is intended to be illustrative, not limiting. Various modi- 
fications may be made without departing from the spirit 
and scope of the invention as defined in the amended 
claims. Modifications and alterations will occur to others 
upon reading and understanding this specification; 
therefore, it is intended that all such modifications and 
alterations are included insofar as they come within the 
scope of the appended claims or equivalents thereof. 

Claims 

1. A microprocessor implemented method for auto- 
matically defining how a plurality of digital images 
should be arranged on a page comprising the ap- 
plication of optimization techniques which evaluate 
and format said plurality of digital images on contact 
sheet-like templates whereon each image is auto- 
matically scaled and positioned to minimize the 
amount of white space on the page thereby produc- 
ing an arrangement of said plurality of images that 
are as large as possible, yet fit within a pre-desig- 
nated page area. 

2. The method of claim 1 wherein optimization is de- 
termined under the following constraints: 

a) rectangles representing each of said images 
do not overlap, 
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b) a bounding box for said rectangles has a min- 
imum area, and 

c) the aspect ratio of the bounding box is a fixed 
value. 

3. The method of claim 2 wherein said bounding box 
has minimum white space and is then formed for 
placement of said images represented by said rec- 
tangles and the aspect ratio of the print area of the 
paper is determined whereby said images can be 
scaled by the appropriate amount to make the 
bounding box the same size as the printable area 
of said page. 

4. A microprocessor implemented method for auto- 
matically generating the digital equivalent of a pho- 
tographers contact print sheet onto which a plurality 
of segmented images will be scaled and distributed 
such that each of said images on a resulting page 
representing said contact print sheet is scaled as 
large as possible and wherein there is a minimum 
of white space remaining on the rendered page, 
wherein said images are scaled by the same 
amount, i.e., the relative size of each segmented 
image stays the same and an optimum spatial dis- 
tribution of the images, represented by rectangles, 
is determined under the following constraints: 

a) rectangles representing each of said images 
do not overlap, 

b) a bounding box for said rectangles has a min- 
imum area, and 

c) the aspect ratio of the bounding box is a fixed 
value. 

5. The method of claim 4 wherein said bounding box 
has minimum white space and is then formed for 
placement of said images represented by said rec- 
tangles and the aspect ratio of the print area of the 
paper is determined whereby said images can be 
scaled by the appropriate amount to make the 
bounding box the same size as the printable area 
of said page. 

6. A image layout processing device comprising: 

a microprocessor programmed to execute an 
image layout template algorithm consisting of 
rules for producing a layout of a plurality of dig- 
ital images in a logical structure wherein each 
image is scaled and positioned to minimize 
white space by uniformly enlarging images to 
fit within said logical structure; and 
a memory for retaining said image layout tem- 
plate algorithm. 
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a means for gang scanning a plurality of images 

into said system; and 

a means for rendering said layout. 

The invention of claim 6 wherein said microproces- 
sor automatically generates templates which define 
how different images should be arranged on a page 
through the application of standard optimization 
techniques having cost functions tuned to the needs 
of SmartPlaten technology. 

The invention of claim 7 wherein a collection of dig- 
ital images are arranged on a page representing a 
contact sheet-like templates whereon each image 
is scaled and positioned to minimize the amount of 
white space on said page and wherein resulting im- 
ages are as large as possible, yet fit within a pre- 
designated area of said page. 



The document processing device of claim 6, further 
comprising: 
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